Marketing Research

MKTG 440

Questionnaire Design

Thoughts on SB LX?

Housekeeping

  • I will give feedback on GA1 before Wednesday
  • GA2: Plan to finalize logistics around implementing the focus group this week (recruiting, identifiying a time/place, etc.)
  • Questions?

Calendar

Mon
Tue
Wed
Thu
Fri
Sat
Sun
2/9Survey
Design
2/10
2/11Survey
Design
2/12
2/13
2/14
2/15
2/16Sampling
2/17
2/18Sampling
2/19
2/20Thinking
Forward
2/21
2/22GA2 due
2/23Exam
Review
2/24
2/25Exam 1
2/26
2/27
2/28
3/1GA3 due
3/2Technology
Day
3/3
3/4No Class
3/5
3/6
3/7
3/8
3/9No Class
3/10
3/11No Class
3/12
3/13
3/14
3/15
  • Will discuss TF more on Wednesday
  • Exam prep materials next week

Agenda

  • Common problems with survey questions
  • Measurement considerations
  • Getting the best answers
  • Survey structure

😀

Formulation biases (wording sensitivity)

Ambiguity

Which brand of shampoo do you use?
Question you want participants to answer Question you let them answer

Resolve the ambiguity

Which brand of shampoo do you use?

More specific:

Which brand or brands of shampoo have you personally used at home during the last month? If you have used more than one brand, please list all the brands that apply.

Specify who, what, when, and where.

Which brand of shampoo do you use?

The Ws Defining the question
Who The respondent
• It is not clear whether the question relates to the individual respondent or the respondent’s total household.
What The brand of shampoo
• It is unclear how the respondent is to answer this question if more than one brand is used.
When Unclear
• The time frame is not specified. The respondent could interpret it as the shampoo used this morning, this week, or over the past year.
Where Not specified
• At home, at the gym, on the road?

Leading questions

Retail experts believe that all consumers should comparison shop. Do you agree?

Agree

Disagree

A leading question introduces bias by suggesting desired answers.

To what extent do you agree or disagree that all consumers should comparison shop?

Strongly
Disagree

Somewhat
Disagree

Neutral

Somewhat
Agree

Strongly
Agree

Loaded questions

Should Americans buy imported automobiles that take away American jobs?

Yes

No

A loaded question suggests a socially desirable answer or involves an emotionally charged issue.

Please select the number that best expresses your tendency when shopping for a car:
Buy Domestic
Cars

1

2

3

4

5

6

7
Buy Foreign
Cars

Leading vs. loaded questions

Leading questions encourage or guide the respondent towards a desired answer.

5 stars across the board means we are doing a good job. How would you rate your service today?

1

2

3

4

5

Loaded questions assume something about the respondent or the consequences of the response.

Have you stopped speeding on the highway?

Yes

No

Double-negatives

Do you oppose not allowing the passage of Proposition 418?

Yes

No

Two “wrongs” don’t make a “right” in surveys.

Do you support or oppose Proposition 414?

Support

Oppose

Measurement issues

Nominal issues

Not collectively exhaustive

Which kind of computer do you own?

Windows PC

Apple

A nominal scale should have a category for every possible response.

Which kind of computer do you own? Check all that apply.

I do not own
a computer

Windows PC

Apple Mac

Other

Double-barreled questions

Do you think Sprite is a tasty and refreshing soft drink?

Yes

No

A double-barreled question asks about two things at once, making responses ambiguous.

Do you think Sprite is a tasty soft drink?

Yes

No
Do you think Sprite is a refreshing soft drink?

Yes

No

Ordinal issues

Ambiguous responses

In a typical month, how often do you shop in department stores?

Never

Occasionally

Sometimes

Often

Regularly

Better:

In a typical month, how often do you shop in department stores?

Less than
once

1 or 2
times

3 or 4
times

More than
4 times

Ambiguous responses

How many classes are you taking this semester?

4

5

6

Other

Better:

How many classes are you taking this semester?

3 or fewer

4

5

6

7 or more

Interval/ratio issues

Bad intervals

I am excited for Spring Break.

Strongly
Disagree

Disagree

Neutral

Strongly
Agree

Use pre-configured Likert Scales (remember that equal distance is an assumption):

I am excited for Spring Break.

Strongly
Disagree

Somewhat
Disagree

Neutral

Somewhat
Agree

Strongly
Agree

Bad intervals

How much would I have to pay you to take your phone away for 24 hours?
$0
$20

Ratio scales should have a range that captures the full distribution of responses.

How much would I have to pay you to take your phone away for 24 hours?
$0
$400

Getting the “best” answers (most accurate and reliable)

Three key questions:

  1. Do we care what they have to say?
  2. Do they know the answer?
  3. Are they willing to tell us the answer?

Each suggests a different reason for why a respondent might be unable or unwilling to answer a survey question.

Inability/unwillingness to answer

Inability/unwillingness to answer Is the respondent qualified? Do we care what they have to say? Can the respondent remember? Do they know the answer? Information may be sensitive Are they willing to tell us the answer?

Is the respondent qualified?

Survey Responses Responses you care about Only let these people finish taking your survey

Qualifications

Screening questions (filter questions) prevent unqualified respondents from being included in the study.

Sometimes combined with a “skip question”, used if the next question (or set of questions) should be responded to only by respondents who satisfy a previous condition.

Are you 21 years of age or older?
○ No → (Interviewer: Ask if someone 21+ is available, or end call)
○ Yes → (Interviewer: Continue)

Inability/unwillingness to answer

Inability/unwillingness to answer Is the respondent qualified? Do we care what they have to say? Can the respondent remember? Do they know the answer? Information may be sensitive Are they willing to tell us the answer?

Memory aids

Answering questions requires effort. Recall is effort. Writing is effort.

Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store.
In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store.
1. Women's dresses
2. Men's apparel
3. Children's apparel
4. Cosmetics
...
16. Jewelry
17. Other (please specify)

Reduced cognitive load

Can the respondent remember?

How many gallons of soft drinks did you consume during the last four weeks?
How often do you consume soft drinks in a typical week?

Less than
once a week

1 to 3 times
per week

4 to 6 times
per week

7 or more
times per week

Note: you can also measure soft drinks in bottles/cans.

Do they remember?

Survey Responses Responses you care about Memory aids Low cognitive load

Unstructured vs. structured questions 1

What hobbies or activities do you engage in at home?

Structured (multiple choice):

Which of the following activities do you consider to be your hobbies?
☐ Gardening
☐ Home repairs
☐ Athletics
☐ Painting
☐ Reading
☐ Other (Specify) ________

Unstructured vs. structured questions 2

What hobbies or activities do you engage in at home?

Structured (dichotomous):

Do you engage in gardening at home?

Yes

No

Unstructured vs. structured summary

Unstructured (Open Ended) Structured (Closed-Ended)
Examples • Sentences
• Unaided recall
• Multiple choice
• Dichotomous
• Scaled response
Pros • Rich array of information
• Good for exploratory research when we are unsure what specific questions to ask
• Easier for respondent to answer
• Easier to code, tabulate, and interpret data
• Less interviewer error
Cons • Burdensome for respondent
• Not good for self-administered surveys
• Difficult to analyze
• Less flexible than open-ended (not good for exploratory research)

Inability/unwillingness to answer

Inability/unwillingness to answer Is the respondent qualified? Do we care what they have to say? Can the respondent remember? Do they know the answer? Information may be sensitive Are they willing to tell us the answer?

Sensitive questions

Respondents may be unwilling to disclose, or unwilling to accurately disclose, sensitive information because this may cause embarrassment or threaten the respondent’s prestige or self-image.

"Binge Drinking" has been defined as consuming five or more alcoholic drinks in one sitting. Have you ever been a binge drinker?

Yes

No
Have you ever consumed five or more drinks in one sitting? For the purposes of this study, a drink is defined as a bottle of beer, a glass of wine, a wine cooler, a shot glass of liquor, a mixed drink, or a similar drink containing alcohol.

Yes

No

Projective techniques: third-person

Thinking about last Friday, how many alcoholic drinks did you consume?
○ None
○ 1 drink
○ 2-3 drinks
○ 4-6 drinks
○ More than 6 drinks
On a typical Friday night, approximately how many alcoholic drinks do you think your college classmates usually consume?
○ None
○ 1 drink
○ 2-3 drinks
○ 4-6 drinks
○ More than 6 drinks

Hiding the true nature of the survey

In a study about gambling and drinking behavior in casinos, the researchers want to identify the participants’ gambling habits.

Have you gambled in the past two months?

Yes

No
Only 1% answered "Yes"
Which of the following things have you done in the past two months?
☐ Gone for a bike ride
☐ Gone to a movie
☐ Gone out for dinner
☐ Gambled — 7% answered "Yes"
☐ Gone to the beach

Will they tell you?

What respondents are willing to share Responses you care about Hide true nature Desensitize questions Projective techniques

Overcoming unwillingness to answer

  • Preface the question with a statement that the behavior of interest is common.
  • Explain why the data are needed.
  • Ask the question using a projective technique.
  • Provide response categories rather than asking for specific figures (e.g., income group instead of income in $).
  • Place sensitive topics at the end of the questionnaire.
  • Avoid “complex questions” or “double negative questions”.

It depends on how you ask

How do Americans feel about kneeling during the national anthem as an act of protest?

Answer: It depends on how you ask.

Source: fivethirtyeight

What’s in a name?

Obamacare vs. Affordable Care Act

Source: cnbc

Try some out…

  • In-App Surveys
  • Email Surveys
    • “Survey”
    • “Feedback”
    • “Experience survey”
    • “Customer experience”
    • “SMG”, “Clarion”, …
  • Receipts

Questionnaire structure

Questionnaire layout summary

  • Introduction
  • Screening questions/screeners/filters to identify qualified respondents
  • Research question section
    • General to specific (funnel approach)
    • Create sections of similar content
    • Some sensitive questions can be placed at the end of this section
  • Last section
    • Demographics (e.g., money, family life): lower the anchoring effect, lower attrition at the beginning
    • Sensitive topics (e.g., political and religious beliefs, involvement in accidents or crimes)

Introduction

  • “Introduce” the topic that is being studied
  • Outline expectations
    • Purpose
    • Expected time
  • “Legal” language/consent
  • Mention incentives (reward for finishing)

Example: Questionnaire flow

Location Type Examples Rationale
Screeners Screening questions Do you own a bicycle? To identify target respondents.
First few Warm-ups What brand bicycle do you own? To ensure the survey is simple and easy to answer. Establish a rapport.
First third Transitions What features do you like best about your bicycle? General questions.
Middle Difficult and complicated questions Please rate your bicycle on each of the following dimensions. The respondent has committed to completing the questionnaire.
Last few Demographic questions What is the highest level of education you have attained? Avoid making respondent feel uneasy early.

Question ordering exercise

Consider these two questions:

  • Question A: “In selecting a department store, how important is convenience of location?”

  • Question B: “What considerations are important to you in selecting a department store?”

Should A or B come earlier in the questionnaire? Why?

Order effects

An order effect is a systematic change in survey responses caused by a question’s position within the survey.

Why it happens:

  • Learning: answering one question can change how respondents interpret and answer subsequent questions
    • Context/priming: earlier questions change what’s “top of mind”
    • Anchoring: early numbers/ideas become a reference point
  • Fatigue: later questions get less attention
    • Place most important questions in the middle

Incentives

  • Incentives are offers to do something for the respondent to increase the probability that the respondent will participate in the survey.

  • Can be monetary or non-monetary

  • Non-monetary incentives:

    • Anonymity: respondent name not identified
    • Confidentiality: respondent’s name is known by the researcher but not divulged to a third party
    • SHORT!!: “The survey will take less than five minutes to complete.”

Test. Test. Test. Finalize.

  • Start with a dry run of the survey on a small, representative set of respondents (15-30)
  • Can also use “bots” to generate test responses, but you ALWAYS want human feedback as well
    • On Qualtrics, go to Tools → Generate test responses
  • Goal: Reveal errors before the survey is launched
  • The longer and more complex the questionnaire, the greater the need for extensive pretesting

Checklists

Conventional wisdom for survey questions (Krosnick)

Best practices for question wording:

  • Use simple, familiar words (avoid technical terms, jargon, and slang)
  • Use simple syntax
  • Avoid words with ambiguous meanings, i.e., aim for wording that all respondents will interpret in the same way
  • Strive for wording that is specific and concrete (as opposed to general and abstract)
  • Make response options exhaustive and mutually exclusive
  • Avoid leading or loaded questions that push respondents toward an answer
  • Ask about one thing at a time (avoid double-barreled questions)
  • Avoid questions with single or double negations

Conventional wisdom for survey structure (Krosnick)

Best practices for survey structure:

  • Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher
  • Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview
  • Questions on the same topic should be grouped together
  • Questions on the same topic should proceed from general to specific
  • Questions on sensitive topics that might make respondents uncomfortable should be placed at the end of the questionnaire
  • Filter questions should be included, to avoid asking respondents questions that do not apply to them

Examples from my inbox

Let’s look at how professional organizations structure their surveys:

  • HBO Max customer satisfaction survey
  • Ad Age subscription cancellation survey

These examples illustrate the principles we’ve discussed.

Max

Max: Introduction (& incentives)

Max: Relevance check

Max: Easy general questions

Max: Easy general questions

Max: More detailed questions

Max: More detailed questions

Max: More detailed questions

Max: Consumer characteristics

Max: Consumer characteristics

Max: Consumer preferences

Max: Demographics

Max: Demographics

Max: Thank you and submit

Ad Age: Email introduction

Ad Age: Cancellation reasons

Ad Age: Likelihood to resubscribe

Ad Age: Resubscribe timeline

Ad Age: Moving to specific Qs

Ad Age: More detailed questions

Ad Age: More detailed questions

Ad Age: More detailed questions

Ad Age: Back to easy

Ad Age: Back to easy

Ad Age: Back to easy

Ad Age: Demographics

Ad Age: Thank you

For next week

  • GA2
  • TF prep